F/G  5/9 


AD-A056  686  BOLT  BERANEK  AND  NEWMAN  INC  CAMBRIDGE  MASS 

DIAGNOSING  STUDENTS*  MISCONCEPTIONS  IN  CAUSAL  MODELS. (U) 
JUN  78  A L STEVENS.  A COLLINS.  S GOLDIN  N00014-76-! 

BBN-3786 


-0083 

NL 


UNCLASSIFIFO 


Report  No.  3786 


Diagnosing  Students’  Misconceptions  in  Causal  Models 

Albert  L.  Stevens,  Allan  Collins,  and  Sally  Goldin 


Prepared  for: 

Office  of  Naval  Research 
Advanced  Research  Projects  Agency 


DIAGNOSING  STUDENTS’  MISCONCEPTIONS  IN  CAUSAL  MODELS 


T — 
\. 

V. 

it 

[ 

1 

I 

f 

I . 


Albert  L.  Stevens 
Allan  Collins 
Sally  Goldin 


Bolt  Beranek  and  Newman  Inc. 
Cambridge,  Massachusetts  02138 


Contract  No.  N00014-76-C-0083,  effective  September  15,  1975. 
Expiration  Date:  September  30,  1978 
Total  Amount  of  Contract  - $337,000 

Principal  Investigator,  Allan  M.  Collins  (617)  491-1850 

Sponsored  by: 

Office  of  Naval  Research 
Contract  Authority  No.  NR  154-379 
Scientific  Officers:  Dr.  Marshall  Farr  and 

Dr.  Henry  Halff 

and 

Advanced  Research  Projects  Agency 
ARPA  Order  No.  2284,  Amendment  4 
Program  Code  No.  61101E 


► 


The  views  and  conclusions  contained  in  this  document  are  those  of  the  authors 
and  should  not  be  interpreted  as  necessarily  representing  the  official 
policies,  either  expressed  or  implied,  of  the  Advanced  Research  Projects 
Agency,  the  Office  of  Naval  Research,  or  the  U.  S.  Government. 

Approved  for  public  release;  distribution  unlimited.  Reproduction  in  whole 
or  in  part  is  permitted  for  any  purpose  of  the  United  States  Government. 


Vo 


U t 


UNCLASSIFIED 


i 


! 


i 


n 
n: 


c 


i: 


H/igifc  N/*  37  ?(o 


SECURITY  CLASSIFICATION  OF  THIS  PAOE  (»*«l  Dm*  Enteied) 


r. 

v t 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM  . 

T.  REPORT  NUMBfck  2.  GOVT  ACCESSION  NO. 

Technical  Report  No.  4 Cq 

1.  RECIPIENT’*  CATALOG  NUMBER 

Diagnosing  Students'  Misconceptions  in 
Causal  Models.  — — — - — - 

Semi-Annual  JFfechnical  /ep«rt 
SeptH|V77-l  Mar  J§78  , r- 

'i 


BBN 


‘ 3786 1 


7.  AUTHOR**) 


%.  CONTRACT  OH  GRANT  NUMBER**) 


Albert  L. /stevens  • Sally/Goldin  / / iN00O14-76-C-afl383  fr 

Allan/Collins  J 1 J -~LJ 


Allan/Collins 

t.  PERFORMING  OROANIIATION  NAME  ANO  AOOACSS 

Bolt  Beranek  and  Newman  Inc/ 

50  Moulton  St. , Cambridge,  MA  02138 


10  PROGRAM  ELEMENT.  PROJECT,  TASK 
AREA  r 


61153 


A 4j»QPri  UH.IT  fi,UMB|"S 
0(d  /RR042':D4/o1 
‘NRrS-T79 


II.  CONTROLLING  OFFICE  NAME  ANO  AOORESS 

Personnel  and  Training  Research  Programs 


Office  of  Naval  Research  (Code  458) 

22211. 


Arlington.  ...  

14.  MONITORING  AOENCY  NAME 


JZA. 


amT»3Z 

Ui! 


>/ 1 


UV'ica) 


IS.  SECURITY  CLASS,  (al  the  report) 


UNCLASSIFIED  ^ _ 


1S«.  OC  CL  ASSISI  cation/ do' 
SCHEOULE 


<>e  I 


Is.  DISTRIBUTION  STATEMENT  (of  Ihlm  Report) 

APPROVED  FOR  PUBLIC  RELEASE  DISTRIBUTION  UNLIMITED. 


17.  DISTRIBUTION  STATEMENT  (ol  the  mbettmct  on  tend  In  Black  20,  II  dllltrent  ham  Report) 


\ 


IS.  SUPPLEMENTARY  not  Cl 

To  appear  in  R.  Sro 


Learning  5hd  Instru 


X 


P.  Federico  and  W.  Montague  (Eds.),  Aptitude, 
on:  Cognition  Process  Analysis. 


If  Key  BOROS  (Continue  on  rim  to  tide  llnactttary  and  Identity  by  black  number) 


Education,  Tutoring,  Computer-Assisted  Instruction,  Learning,  Teaching, 
Educational  Psychology. 


20  ABSTRACT  (Continue  on  rororoo  •!<#«  If  r 


•omry  mtd  identity  by  block  number) 


Tutorial  dialogues  can  be  analyzed  as  an  interaction  in  which  a tutor 
"debugs"  a student's  knowledge  representation  by  diagnosing  and  correcting  ^ 
conceptual  misunderstandings.  In  this  paper^-we-emtlitwf  some  tentative 
steps  toward  a theory  which  describes  tutorial  interaction^  'jW^Outlinet  the 
goal  structure  of  a tutor,  describe^types  of  conceptual  bugs  that  students 
have  in  their  understanding  of  physical  processes  and  discuss*  some  of  the 
representational  viewpoints  necessary  to  diagnose  and  correct  these  bugs.^ 


- I 


nn  form 

w I JAM  72 


1473 


tOlTION  OF  1 NOV  SS  IS  OBSOLETE 


\ 


UNCLASSIFIED 

SECURITY  CLASSIFICATION  of  THIS  PAOE  (When  Date  entered) 


0 60  X 0 / 


Ms< 


j. 


r 


] 


3 


i 

r 

i 

i: 

j - 

I. 

i: 

i: 


Diagnosing  Students'  Misconceptions  in  Causal  Models 


Albert  L.  Stevens 
Allan  Collins 
Sally  Goldin 


( 


Bolt  Beranek  and  Newman  Inc. 
50  Moulton  Street 
Cambridge,  Mass.  02138 


2 


i 


1 

i 


□ 


Abstract 


Tutorial  dialogues  can  be  analyzed  as  an 
interaction  in  which  a tutor  "debugs"  a student's  knowledge 
representation  by  diagnosing  and  correcting  conceptual 
misunderstandings.  In  this  paper,  we  outline  some  tentative 
steps  toward  a theory  which  describes  tutorial  interactions. 
We  outline  the  goal  structure  of  a tutor,  describe  types  of 
conceptual  bugs  that  students  have  in  their  understanding  of 
physical  processes  and  discuss  some  of  the  representational 
viewpoints  necessary  to  diagnose  and  correct  these  bugs. 


Diagnosing  Students'  Misconceptions  in  Causal  Models 

Albert  Stevens 
Allan  Collins 
Sally  Goldin 

We  are  building  a computer  aided  instructional  system 
which  tutors  students  to  reason  about  and  understand 
physical  processes.  In  order  to  build  such  a system,  we 
have  been  forced  to  confront  several  fundamental  issues 
about  the  tutorial  process: 

(1)  What  is  the  goal  structure  that  governs  a tutor's 
selection  of  examples,  questions  and  statements  at 
different  points  in  the  dialogue? 

(2)  What  are  the  types  of  misconceptions  that  students  have 
and  how  do  tutors  diagnose  misconceptions  from  errors 
students  make? 

(1)  What  are  the  abstractions  and  viewpoints  that  tutors 
use  to  explain  physical  processes? 

We  believe  we  are  takinq  tentative  steps  toward  a 
theory  which  addresses  these  issues.  Our  approach  is  to 
work  out  a theory  based  on  analyses  of  tutoring  dialogues 
and  experiments  and  then  build  a system  based  on  that 
theory.  Building  the  system  reveals  points  where  the  theory 
is  inadequate  or  wrong.  In  subsequent  iterations  of  this 
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process,  we  concentrate  on  these  weak  points.  In  this 
paper,  we  will  describe  the  first  version  of  our  system,  its 
weak  points  and  the  steps  in  analysis  and  theory  development 
we  are  taking  to  remedy  these  weak  points. 


The  WHY  System 


The  first  version  of  our  system,  called  the  WHY  system, 
can  carry  on  a simple  teaching  dialogue  about  the  causes  of 
rainfall.  The  theory  on  which  the  WHY  system  is  based  is 
formalized  as  a set  of  production  rules  representing 
teaching  strategies  (Collins,  1977)  and  a script-like 
knowledge  structure  (Schank  & Abelson,  1977).  The  script 
structure  represents  the  different  temporal  and  causal  steps 
in  processes  that  affect  rainfall.  Many  scripts  in  the 
system  can  be  decomposed  into  more  detailed  subscripts.  The 
resulting  embedded  structure  is  used  to  represent  levels  of 
detail  of  knowledge  about  different  processes. 


Figure  1 illustrates  these  aspects  of  knowledge 
organization . It  shows  the  top-level  script  for  heavy 
rainfall  which  consists  of  four  steps:  Evaporation,  movement 
of  the  air  mass,  cooling  and  precipitation.  The  subscript 
for  the  first  step  is  also  shown  and  consists  of  a more 
detailed  breakdown  of  the  steps  involved  in  evaporation. 
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Heavy  Rainfall 
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The  teaching  strategy  rules  are  stated  in  terms  of  a 
conditional  test  of  the  student’s  response  to  a question 
paired  with  an  action  to  perform  if  the  test  is  true. 


Example  rules  are: 

(1)  I_f  the  student  gives  as  an  explanation  a factor  that 

is  not  an  immediate  cause  in  the  causal  chain, 
then  ask  for  the  intermediate  steps. 

(2)  Tf_  the  student  gives  as  an  explanation  one  or  more 

factors  that  are  not  necessary, 
then  formulate  a general  rule  by  asserting  that  the 
factor  is  necessary  and  ask  the  student  if  the 
rule  is  true. 

The  theory  in  Collins  (1976)  consists  of  twenty-four  rules. 
The  first  version  of  the  Why  system  contains  seven  rules 
which  test  for  missing  script  steps,  ask  students  questions 
and  present  information. 

Problems  With  the  Current  System 

The  current  WHY  system  is  able  to  carry  on  simple 
tutoring  dialogues  about  the  causes  of  rainfall.  It  can  ask 
questions  about  places  where  heavy  rainfall  occurs,  diagnose 
missing  steps  in  the  student's  knowledge  and  inform  the 
student  about  the  correct  steps.  A bit  of  interaction  with 
the  system  reveals  several  problems:  (1)  It  has  little 
global  perspective  about  the  dialogue  and  thus  bases  its 
questions  and  responses  almost  exclusively  on  local  context. 
(2)  It  is  sensitive  to  student  errors,  but  typically  misses 
the  cause  of  these  errors,  correcting  the  surface  error  but 
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failing  to  diagnose  the  underlying  misconception  that  the 
error  reflects.  (3)  There  are  many  important  aspects  of 
physical  processes  and  many  important  ways  of  describing 
physical  processes  that  the  WHY  system  fails  to  use. 

These  problems  parallel  the  issues  listed  earlier.  The 
lack  of  global  perspective  in  the  current  system  arises 
because  there  is  no  coherent  goal  structure.  The 
teaching-strategy  rules  we  originally  developed  are  based  on 
the  students'  immediate  responses  and  have  no  way  to 
establish  and  be  influenced  by  higher  level  goals. 

The  types  of  misconceptions  in  students'  knowledge  that 
a system  can  diagnose  are  heavily  dependent  on  the  knowledge 
represented  in  the  system.  The  script  structures  in  the 
current  WHY  system  are  able  to  represent  m i sconcept ions 
which  result  because  of  missing  substeps  or  extra  substeps 
in  the  various  scripts.  These  are  only  two  of  several  types 
of  misconceptions  that  occur.  We  will  discuss  others  below. 


The  problems  of  failing  to  discuss  important  aspects  of 
physical  processes  and  failing  to  use  important  ways  of 
describing  physical  processes  arise  because  the 
script-subscript  formalism  is  limited.  We  believe  that 
representing  knowledge  about  physical  processes  requires 
multiple  "representational  viewpoints."  Our  script 
structures  provide  one  of  those,  the  viewpoint  of  a sequence 
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of  temporally  ordered  processes,  some  causally  related  to 
others,  and  some  subprocesses  of  others.  This 
representational  viewpoint  is  important,  but  equally 
important  is  the  "functional  viewpoint”  which  emphasizes  the 
functional  relationships  among  attributes  of  the  various 
objects  involved  in  different  processes. 

In  the  following  sections,  we  will  discuss  each  of 
these  problems  and  provide  some  initial  ideas  about  a theory 
necessary  to  deal  with  them.  It  will  be  apparent  that  they 
are  all  intimately  interrelated  and  that  a key  element 
necessary  for  their  solution  is  an  adequate  formalism  for 
representing  the  knowledge  taught. 

Some  Proposed  Solutions 

Goal  Structure.  One  of  the  major  constraints  on  a 
theory  of  tutoring  is  that  it  should  adequately  describe  the 
structure  of  tutorial  dialogues.  Our  analyses  of  tutorial 
dialogues  reveals  a general  structure  that  follows  from  the 
script  structure  of  the  knowledge  base.  Tutors  discuss 
topics  in  a rational  order,  typically  following  the 
discussion  of  one  process  with  discussion  of  a temporally  or 
causally  adjacent  process  or  with  a discussion  of  component 
subprocesses . 


Detailed  dialogue  structure  is  much  more 
problematical.  Close  examination  of  tutorial  dialogues 
reveal  that  the  tutor  probes  the  student  about  many 
different  aspects  of  the  knowledge.  When  the  student  makes 
an  error,  the  tutor  will  sometimes  correct  it  immediately, 
but  in  many  cases  will  ask  other  questions  until  the 
misconception  underlying  the  error  is  isolated.  The 
treatment  of  the  misconception  may  then  require  a number  of 
interchanges  during  which  the  tutor  tests  the  student's 
knowledge  and  supplies  the  relevant  information.  We  believe 
that  a powerful  perspective  from  which  to  view  this  more 
detailed  structure  of  tutorial  dialogues  is  that  of  the 
tutor  as  a "debugger."  Much  of  the  detailed  structure  of  a 
dialogue  results  from  the  tutor  using  various  strategies  to 
diagnose  students'  misconceptions,  or  "bugs,"  and  then 
applying  strategies  to  correct  them. 

In  order  to  investigate  this  perspective  of  tutors  as 
debuggers,  we  have  conducted  dialogues  where  the  questions 
and  responses  were  communicated  over  linked  terminals  and 
where  the  tutors  verbally  commented  on  two  aspects  of  the 
dialogue  as  they  proceeded.  The  two  aspects  were:  (1)  What 
they  thought  the  student  knew  or  didn't  know,  based  on  the 
student's  response,  and  (2)  why  they  responded  to  the 
student  in  the  way  they  did.  This  technique  supplies  data 
normally  unavailable  for  a dialogue  analysis,  providing 
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insights  into  how  the  tutor  develops  a model  of  the  student, 
how  the  tutor  organizes  the  knowledge  being  taught  and  how 
these  two  factors  influence  the  tutors  choice  of  questions 
and  responses  to  the  student. 

Using  this  data,  we  developed  the  outlines  of  a theory 
of  tutors'  goal  structures.  The  goal  structure  we  derived 
is  summarized  in  Table  1.  The  top  level  goals  are  (1) 
refine  the  student's  causal  model  and  (2)  refine  the 
student's  procedures  for  applying  the  model.  These  directly 
govern  the  selection  of  cases.  As  the  student's  knowledge 
becomes  more  refined,  moving  from  an  understanding  of 
first-order  factors  to  higher-order  factors,  cases  are 
selected  which  are  exemplary  of  the  factors  the  tutor  is 
trying  to  teach.  As  the  student's  predictive  ability 
becomes  refined,  cases  are  selected  which  are  progressively 
more  novel  and  complex,  taxing  the  student's  predictive 
ability  more  and  more. 

The  process  of  achieving  these  top-level  goals  involves 
two  types  of  subgoals:  diagnosis  and  correction.  Both  of 
these  subgoals  govern  the  selection  of  basic  strategies. 

The  purpose  of  diagnosis  is  to  discover  gaps  and 
misconceptions  in  the  student's  knowledge.  This  generally 
requires  that  the  tutor  probe  the  student  by  asking  for 
relevant  factors,  by  requiring  the  student  to  make 
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Table  1 


Outline  of  a Socratic  Tutor’s  Goal  Structure.  The  manifestations  refer  to 
the  rules  described  in  Collins  (1976)  and  Stevens  and  Collins  (1977). 


Goal  s 


Manifestation 


Refine  the  student's  causal  model  Case  selection  rules: 

moving  from  1st  to  nth  order  factors.  Select  cases  that  are 

exemplary  of  the  relevent 
f actor . 


Refine  the  student's  procedures  for 
applying  the  causal  model  to  novel 
cases . 


Case  selection  rules: 

Select  less  familiar  cases, 
exemplary  of  new  factors. 


Subgoal s 

Diagnose  the  student's  "bugs", 

(i.e.  the  difference  between  the 
student's  knowledge  and  the  tutor's 
knowl edge  . ) 

Correct  the  diagnosed  bugs 


Ask-for-factor  rules. 
Prediction  rules. 

Entrapment  rules. 
Probe-reasoning-strategy  rules 

Inform-student  rules 
Missing- fac tor  rules 
Forming  hypotheses  rules 
Testing  hypotheses  rules 
Information-collection  rules 


predictions  about  carefully  selected  cases,  and  by  trying  to 
entrap  the  student  into  making  incorrect  predictions.  It  is 
clear  from  our  analysis  of  human  dialogues  that  diagnosis 
cannot  be  completely  characterized  in  terms  of  a simple 
mapping  between  students'  errors  and  their  conceptual  bugs. 
Rather  the  process  involves  sophisticated  use  of  a student 
model  and  knowledge  about  common  bugs  in  order  to  simulate 
the  student's  reasoning  processes  and  pinpoint  the 
underlying  conceptual  errors  or  missing  information.  In 
some  cases,  a single  answer  may  reveal  a whole  set  of  bugs, 
while  in  other  cases,  the  tutor  must  carefully  probe  the 
student,  testing  alternative  hypothesized  bugs  to  reveal  the 
misconception . 

Typically,  when  a conceptual  bug  is  diagnosed,  the 
tutor  attempts  to  correct  it.  This  may  require  a single 
statement  for  simple  factual  errors  or  an  extended  dialogue 
to  correct  problems  in  the  student's  causal  model.  In 
Stevens  and  Collins  (1977)  we  illustrate  the  application  of 
this  goal  structure  model  by  analyzing  a tutorial  dialogue. 

Our  outline  of  goal  structure  is  relatively  general  and 
probably  can  be  applied  to  many  different  knowledge  domains 
and  tutorial  interactions.  However,  in  order  to  specify  it 
in  detail,  we  need  to  know  what  the  bugs  are,  how  they  can 
be  represented,  how  they  are  diagnosed  from  errors  and  how 
they  can  be  corrected. 
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Types  of  bugs  in  understanding  rainfall.  Our  analyses 
of  dialogues  show  that  tutors  spend  a good  part  of  their 
time  diagnosing  conceptual  bugs  from  errors  manifested  in 
the  dialogue.  We  believe  that  one  of  the  major  skills  a 
teacher  possesses  is  knowledge  about  the  types  of  conceptual 
bugs  students  are  likely  to  have,  the  manifestations  of 
these  bugs  and  methods  for  correcting  them.  It  is  thus 
clear  that  an  important  component  of  any  teaching  system  is 
a method  for  representing,  diagnosing  and  correcting  bugs. 

To  examine  the  types  of  bugs  that  occur  in  students' 
understanding  of  rainfall,  we  carried  out  a simple 
experiment.  We  compiled  a systematic  test  about  the  causes 
of  heavy  rainfall  by  generating  questions  for  all  major 
script  nodes  in  the  current  WHY  system  representation  of 
rainfall.  This  included,  for  each  node,  a question  which 
queried  prior  script  steps  and  questions  which  queried 
subsequent  script  steps.  These  questions  were  presented  to 
subjects  on  a questionnaire.  At  the  top  of  the 
questionnaire  was  a context-setting  paragraph  which 
explained  that  all  questions  were  to  be  interpreted  as 
referring  to  areas  of  the  world  where  heavy  rainfall  occurs 
and  described  what  we  meant  by  heavy  rainfall.  Some  typical 
questions  from  this  test  are: 


"How  is  the  moisture  content  of  the  air  related  to 

heavy  rainfall?" 

"What  role  does  rising  air  play  in  causing  rainfall?" 

"What  causes  evaporation?" 

There  were  a total  of  32  questions.  The  questions  were 
initially  randomized  and  each  subject  received  them  in  the 
same  random  order.  The  questionnaire  also  included  two 
questions,  one  which  asked  the  subjects  to  name  areas  of  the 
world  which  have  heavy  rainfall  and  the  other  areas  which 
have  little  rainfall. 

The  instructions  emphasized  that  even  if  the  subjects 
felt  they  did  not  know  an  answer,  they  should  try  to  answer 
the  question.  We  adopted  these  instructions,  because  the 
typical  response  given  by  subjects  when  confronted  with  this 
test  was  "I  don’t  know  anything  about  rainfall."  Subsequent 
probing  revealed  that  they  often  knew  a good  deal  more  than 
they  thought.  The  test  was  administered  to  subjects 
individually  and  typically  was  completed  in  about  30 
minutes . 

The  experiment  provided  us  with  a substantial  body  of 
data  on  errors  and  misconceptions.  In  order  to  analyze  the 
responses  in  detail,  all  answers  that  were  judged  incorrect 
were  tabulated  under  the  appropriate  questions.  We  then 
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analyzed  these  errors  by  developing  a basic  set  of 
conceptual  bugs  and  classifying  the  errors  according  to  this 
set.  This  analysis  revealed  two  points  of  major  interest: 


(1)  A particular  conceptual  bug  is  often  shared  by 
many  students. 

(2)  A particular  conceptual  bug  is  often  manifested  in 
many  different  ways. 

For  example,  a bug  we  call  the  Cooling-by-contact  bug  is 
very  common,  occurring  for  6 of  the  8 subjects.  Some 
verbatim  examples  of  manifestations  of  this  bug  are: 

(1)  "Cold  air  masses  cool  warm  air  masses  when  they 
col  1 ide  . " 

(2)  "Winds  cause  air  to  cool." 

(1)  "Mountains  cause  condensation  because  cold  land 
touching  warm  air  causes  condensation." 

(4)  "Cold  fronts,  wind,  snow  and  rain  cause  air  to 

| 

cool . " 

(5)  "Cold  air  masses  cool  the  clouds  so  the  rain 
falls  ." 

None  of  the  above  types  of  cooling  are  of  any 
consequence  in  causing  heavy  rainfall.  The  type  of  cooling 

-14- 

•J 


necessary  occurs  when  an  air  mass  is  forced  to  rise.  The 
rising  results  in  expansion  and  energy  loss. 


We  identified  a total  of  sixteen  different  bugs  from 
this  experiment.  They  are  shown  in  order  of  frequency  in 
Table  2.  Using  these  sixteen  bugs,  we  were  able  to  account 
for  58%  of  the  answers  originally  judged  to  be  incorrect  or 
omitted.  (By  ignoring  omissions,  we  were  able  account  for 
72%. ) We  are  being  conservative  in  this  accounting.  For 
many  of  the  remaining  errors  and  omissions,  one  can  make  a 
plausible  argument  that  these  bugs  could  lead  to  that  error. 

Many  statements  that  we  did  not  account  for  were  factual 
errors,  for  example,  "Heavy  rainfall  occurs  only  in  warm 
areas."  (Heavy  rainfall  occurs  in  many  cool  and  cold  areas 

4 

of  the  world.)  Others  were  naming  errors.  For  example, 

"When  water  evaporates,  it  turns  to  steam."  (The  standard 
term  in  meteorology  for  the  product  of  evaporation  is  water 
vapo  r . ) 

t 

Many  of  the  bugs  we  observed  are  specific  to  the  domain  - i 

i 

of  rainfall.  This  should  be  neither  surprising  nor 

I 

disturbing.  One  of  the  skills  a good  teacher  must  possess 
is  knowledge  of  the  types  of  misconceptions  that  arise  in 
the  domain  taught.  It  is  likely  that  there  are  other  bugs 
which  occur  in  students'  knowledge  about  rainfall,  but  it 
will  surprise  us  if  this  number  is  unmanageably  large. 
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Table  2.  The  set  of  observed  misconceptions. 


Misconception  Number 

of 

Subjects  Example 

(1 ) 

Cooling-by-contact 

6 

"Mountains  cause  condensation 
because  cold  land  touching  air 
causes  condensation." 

(2) 

Heating-by-radiation 

6 

"The  sun  warms  the  air." 

(3) 

Small- moisture- source 

5 

"A  12  by  12  by  10  foot  pond  is 
enough  to  cause  rainfall." 

(4) 

Rising-causes- 
incr eased -pres sure 

3 

"Rising  air  makes  the  moist  air 
rise,  pressure  increases  ..." 

(5) 

Absorbtion-by- 

expansion 

3 

"...decrease  in  pressure  causes 
water  molecules  to  expand, 
causes  evaporation." 

(6) 

Heat ing-by -con tact 

3 

"...land  warms  the  air  at  night 

(7) 

Squeezing-cause s- 
condensation 

2 

"Putting  pressure  on  air 
masses  causes  condensation." 

(8) 

Temperature-of-water- 

irrelevant-for- 

evaporation 

2 

"Temperature  of  water  is 
unrelated  to  evaporation." 

(9) 

Temperature- 
differential- 
causes- evaporation 

2 

"Air  has  to  be  cooler  than 
the  body  of  water  for 
evaporation  to  occur." 

(10) 

Insuf f ic ient- 

warming-of- 

water 

2 

"A  current  can  be  warm  because 
it  comes  from  a warm  source 
of  water — for  example,  a 
lake  which  is  warm." 

(11  ) 

Heating- causes- 
condensation 

1 

"Air  warming  up  causes  rainfall 

(12) 

Winds-cause- 
pressure- increases 

1 

"Winds  are  forceful  and 
cause  various  air  pressures." 

(13) 

Cool ing-causes- 
evaporation 

1 

"When  a body  of  water  is  cold, 
i t evaporates  . " 

(14) 

Rising-results- in 
pressure-equalization 

1 

"Air  that  is  warmer  is  expanded 
and  has  less  pressure.  It  rises 
until  its  pressure  is  equal  to 
surrounding  air." 

(15) 

Cool ing-cause-air 
to-r ise 

1 

"Cooling  causes  air  to  rise." 

(16) 

Eva po ration-cause s- 
air-to-rise 

1 

"Evaporation  causes  air  to  rise 

To  represent  these  bugs  in  a way  that  makes  it  possible 
for  a teaching  system  with  a d i agnose-and-correct  goal 
structure  to  use  them  is  an  important  step.  In  principle, 
the  current  script-like  formalism  could  be  used.  In 
practice,  such  things  as  incorrect  functional  relationship 
(e.g.  the  Heating-causes-condensation  bug)  or  incorrect 
attributes  (e.g.  the  temperature  of  mountains  in  the 
Cool i ng-by-contact  bug)  seem  to  require  a different 
representational  viewpoint  than  those  provided  by  script 
structures.  We  will  provide  some  steps  toward  a solution 
below  in  the  section  on  representation. 

Explaining  physical  processes.  The  third  problem  we 
described  in  the  introduction  is  the  nature  of  the 
abstractions  and  viewpoints  that  tutors  use  to  describe 
physical  processes.  The  teaching  dialogues  we  have  examined 
require  a multi-leveled  structure,  with  script-like 
knowledge  necessary  to  support  some  parts  of  the  discussions 
and  relatively  low-level  detailed  knowledge  about  physical 
principles  necessary  to  support  other  parts.  More 
interestingly,  to  adequately  support  the  dialogues  requires 
that  the  knowledge  be  factorable  in  several  ways.  Tutors 
discuss  far  more  than  causal  and  temporal  linkages  between 
steps  in  a script  structure.  They  probe  and  discuss 
information  about  attributes  of  the  actors  that  are 
important,  the  results  of  processes  and  the  form  of  the 


functional  relationship  which  holds  between  the  attributes 
of  actors  and  the  results  of  processes. 


Some  examples  of  tutors'  statements  and  questions  are 
shown  in  Table  3.  In  each  case,  the  question  or  statement 
refers  to  one  of  the  specific  aspects  of  the  knowledge  we 
just  described.  A cursory  examination  of  our  dialogues 
suggest  that  a large  percentage  of  tutors'  statements  and 
questions  fall  into  these  categories.  For  example,  in  a 
representative  dialogue  which  consisted  of  41  exchanges 
between  the  tutor  and  student,  four  of  the  tutor's 
statements  were  about  attributes  of  actors,  four  were  about 
results  of  processes  and  seven  were  about  functional 
relationships.  This  accounting  includes  15  of  the  tutor's 
statements.  Of  the  remaining  26,  eight  are  references  to 
prior,  intermediate  or  subsequent  processes  at  a level  of 
abstraction  that  can  be  handled  by  script  structures.  The 
remainder,  which  we  do  not  have  good  ideas  about,  include 
references  to  the  spatial  structure  of  the  processes, 
descriptions  of  physical  principles  and  explication  of  a 
metaphor . 


Representing  the  knowledge  domain 


For  each  of  the  three  problems  we  have  discussed,  a key 
element  necessary  for  its  solution  is  an  adequate  formalism 
for  representing  the  knowledge  taught.  To  specify  the  goal 
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Table  3-  Example  statements  for  each  part  of  the  representation. 


Factors  (Attributes  of  Actors) 


structure  of  a tutor  in  detail  requires  specifying 
m i sconcept ions  and  methods  for  correcting  them.  Specifying 
m isconcept ions  and  the  proper  abstractions  and  viewpoints 
from  which  to  diagnose  and  correct  them  requires  a detailed 
formalism  for  representing  the  knowledge  taught.  Script 
structures  can  be  used  to  represent  ordered  causal  and 
temporal  processes,  but  this  handies  only  a small  number  of 
bug  representations  and  viewpoints  from  which  to  discuss 
them.  In  this  section,  we  will  describe  one  additional 
representational  viewpoint  that  seems  important  for  a 
tutorial  system. 

A representation  of  functional  relationships.  The 
basic  unit  of  our  representation  for  functional 
relationships  is  a description  of  some  process  such  as 
cooling  or  evaporation.  An  example  is  shown  in  Figure  2. 
This  represents  the  process  of  evaporation  as  it  occurs  in 
the  rainfall  domain.  Its  parts  are: 

(1)  A set  of  actors  each  with  a role  in  the  overall 
process.  For  example,  the  ocean  plays  the  role  of 
moisture  source. 

(2)  A set  of  factors  which  affect  the  process.  The  factors 
are  all  attributes  of  actors.  For  example,  the 
temperature  of  the  source  body-o f-water  is  a factor  in 
evapo  ration . 
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Figure  2.  A functional  representation  for  evaporation 


EVAPORATION 
Ac  tors 

Source:  Large-body-of-water 
Destination:  Air-mass 
Factors 

Temperature (Source) 

Temperature (Destination) 

Prox im ity(Sour ce  , Destination) 
Functional-relationship 

Positive(Temperature(Source) ) 

Positive (Temperature (Destination)) 
Positive(Proximity( Source,  Destination)) 
Result 

Increase(Humidity(Destination)  ) 


(3)  A description  of  the  resul t of  the  process.  The  result 
is  always  a change  of  an  attribute  of  one  of  the 
actors.  For  example,  the  result  of  evaporation  is  to 
increase  the  humidity  of  the  destination  air  mass. 

(4)  A description  of  the  functional  relationship  which 
holds  among  the  factors  and  the  result.  We  believe 
there  is  room  for  complexity  and  subtlety  in  the 
description  of  functional  relations,  but  we  currently 
use  a simple  descriptive  scheme  which  allows  positive 
and  inverse  relationships.  For  example,  in  evaporation 
there  is  a positive  relationship  between  the 
temperature  of  the  moisture  source  and  the  resulting 
humidity  of  the  air  mass. 

This  representation  is  general  in  two  ways.  It  can  be 
partially  specified  by  assigning  values  to  the  actor 
attributes.  For  example,  representing  an  instance  of  a 
large  amount  of  evaporation  requires  assigning  values  like 
"warm"  to  the  temperatures  of  source  and  destination  and  a 
value  like  "adjacent"  to  the  proximity  relationship. 
Inference  rules  which  make  use  of  the  information  about 
relevant  attributes  and  functional  relationships  can  be 
constructed  to  check  (at  some  level  of  appr ox imat ion)  if  the 
assigned  values  of  factors  and  result  are  consistent. 
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The  second  way  that  this  representation  can  be  further 


specified  is  by  instantiating  the  actors.  For  example,  in 
the  case  of  rainfall  over  Ireland,  the  source  is  the  Gulf 
Stream . 


Thus,  the  information  in  this  representation  provides  a 
way  for  generating  representations  of  different  amounts  of 
evaporation  and  for  representing  these  different  amounts 
with  different  actors.  This  representation  provides  an 
additional  representational  viewpoint  that  is  missing  from 
script  structures. 


Representing  Bugs.  In  addition  to  representing 
knowledge  that  is  correct,  it  must  be  possible  to  represent 
m i sconcept ions . One  constraint  on  a knowledge 
representation  used  for  teaching  is  to  represent 
misconceptions  as  meaningful  transformations  of  the  basic 
knowledge  representations  (Brown  and  Burton,  1978).  For 
example,  consider  the  representation  for  the 
Absorption-by-expansion  bug  shown  in  Figure  3.  The  key  part 
is  highlighted.  It  consists  principally  of  a substitution 
of  pressure  for  temperature  as  the  relevant  attribute  of  the 
destination  in  the  normal  representation  for  evaporation. 
Representing  bugs  in  the  same  format  as  the  correct 
knowledge  makes  differential  diagnosis  possible.  In  trying 
to  decide  whether  a student  has  the  cooling-by-contact  bug. 
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Figure  3-  The  Absorption-by-expansion  bug. 

Evaporation  under  the  Absorption-by-expansion  bug 
Actors 

Source:  Large-body-of-water 
Destination:  Air-mass 
Factors 

Prt3f?ai^ru(Destination) 

Prox im ity ( Source , Destination) 
Functional-relationship 

Inverse ( Pressure (Destination) ) 
Positive(Pr ox imity( Source,  Destination)) 
Result 

Increased!  urn  id  ity(Destination)  ) 


I 


asking  the  student  what  actors  are  involved  will  not  provide 
any  relevant  information.  Either  winds,  mountains  or  cold 
air  masses  will  still  be  mentioned  as  important.  It  is  the 
role  or  attributes  of  these  actors  that  supplies  the 
leverage.  However,  in  the  Small-moisture-source  bug,  the 
actor  itself  supplies  the  diagnostic  leverage  point. 

Bugs  show  up  in  all  parts  of  the  representation.  For 
example,  the  Cool ing-by-contact  bug  is  represented  as  a 


difference  in  the  role  of  the  object,  or  as  a difference  in 
the  relevant  attribute  of  the  object.  The 
Heating-causes-condensation  bug  is  represented  as  a 
difference  in  functional  relationship.  The 
Small-moisture-source  bug  is  represented  as  a difference  of 
actor  in  the  source  role. 


Remaining  Problems 

In  the  previous  sections,  we  outlined  some  tentative 
steps  toward  solving  the  problems  of  goal  structure, 
representing  misconceptions,  and  providing  the  additional 
representational  viewpoint  of  a set  of  functionally  related 
processes.  We  believe  the  heart  of  these  problems  lies  in 
the  representation  of  knowledge.  Our  tentative  steps  toward 
representing  knowledge  and  misconceptions  about  physical 
processes  extend  the  script-like  representation  we  have  been 
using,  but  we  believe  we  are  still  just  scratching  the 


surface . 
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Adding  this  one  viewpoint  has  given  us  more  windows 
into  problems  that  were  opaque  using  our  previous  notation. 
In  the  remainder  of  this  paper,  we  will  point  out  the  nature 
of  some  of  the  problems  that  we  can  now  see. 

Interacting  Bugs.  In  most  cases,  a single  bug  accounts 
for  each  error,  but  there  are  cases  where  bugs  interact  to 
produce  a single  misconception.  Brown  and  Burton  (19"78) 
have  shown  that  in  arithmetic,  students  often  have  a set  of 
bugs  that  interact  to  produce  non-obvious  patterns  of 
errors.  The  observations  from  our  experiment  suggest  that 
similar  things  happen  in  the  rainfall  domain.  For  example, 
one  subject  said  in  response  to  a question  about  the  role  of 
cold  air  masses  : 

"Cold  air  masses  hitting  warm  air  masses  cause 
condensation . " 

Since  she  mentioned  contact  and  not  rising,  the  most 
straightforward  diagnosis  from  this  statement  is  that  she 
has  a Cooling-by-contact  bug.  However,  the  problem  really 
seems  to  be  due  to  two  interacting  bugs:  the 
Heating-by-contact  bug  and  the  Heating-causes-condensat ion 
bug.  Two  of  this  student's  responses  to  other  questions 
were : 


I 
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"Air  warming  up  causes  rainfall." 

"Tropical  winds  warm  air." 

Thus,  her  description  of  condensation  caused  by  cold  air 
masses  hitting  warm  air  masses  is  most  likely  due  to  these 
bugs  interacting  to  produce  a model  in  which  the  cold  air  is 
warmed  up  from  contact  with  the  warm  air  (the 
Heating-by-contact  bug)  and  this  warming  causes  rainfall 
(the  Heating-causes-condensation  bug). 

The  existence  of  interactions  imply  that  the  mapping 
from  errors  to  bugs  is  not  one-to-one.  We  suspect  that 
there  are  many  cases  where  the  relationship  between  a set  of 
errors  and  the  underlying  bugs  may  be  quite  subtle.  The 
existence  of  non-obvious  interactions  may  account  for  our 
inability  to  classify  many  of  the  errors  we  observed. 

Where  do  bugs  come  from?  Having  looked  at  the  bugs  we 
have  isolated,  we  now  believe  that  they  still  are  relatively 
shallow,  reflecting  even  deeper  levels  of  misconceptions  in 
knowledge.  The  major  reason  for  believing  this  is  that  bugs 
themselves  seem  to  form  patterns.  The  patterns  seem  best 
explained  as  the  result  of  deeper  problems  in  the  student's 
knowledge.  Sometimes  these  deeper  problems  are  due  to  the 
application  of  an  incorrect  metaphor  in  understanding  a 
process;  other  times  the  patterns  reflect  incorrect  or 
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missing  general  relationships  between  process,  like  the 
notion  of  inverse,  or  positive  feedback. 

An  example  of  a pattern  that  reflects  an  incorrect 
application  of  a metaphor  is  the  "sponge  pattern".  It 
includes  two  bugs:  the  Absorption-by-expansion  bug  and  the 
Squeezing-causes-condensation  bug.  In  effect,  the  student 
views  the  air  mass  as  a giant  sponge,  expanding  to  absorb 
moisture  and  later  having  it  squeezed  out.  Tutors  typically 
deal  with  this  deep-level  misconception  by  using  a 
"container"  analogy  for  the  air  mass,  identifying  the 
capacity  of  the  container  with  air  temperature  rather  than 
size. 

A second  type  of  pattern  is  that  which  arises  because 
of  missing  generalizations  about  process  relationships.  For 
example,  the  pattern  which  includes  the 
Heating-causes-condensation  bug  but  which  also  includes  the 
correct  functional  relationship  between  heating  and 
evaporation  seems  to  reflect  the  student's  lack  of 
understanding  that  condensation  and  evaporation  are  inverse 
processes.  Tutors  deal  with  this  bug  by  informing  the 
student  that  the  two  processes  are  inverses  and  explaining 
the  sense  in  which  they  are  inverses. 

These  processes  of  understanding  draw  on  a large  set  of 
real-world  knowledge  that  students  have  built  up  over  their 
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lifetime.  The  bugs  often  seem  to  depend  on  the  student's 
failure  to  understand  some  deep  physical  principles  that 
support  the  correct  model.  In  order  for  tutors  to  deal  with 
conceptual  bugs,  they  must  recognize  this  mode  of 
understanding  and  attempt  to  discover  what  models  the 
student  applies  to  understand  the  processes  taught. 

Other  representational  viewpoints.  The  existence  of 
patterns  of  bugs  implies  that  that  there  are  still  other 
representational  viewpoints  necessary  to  deal  completely 
with  physical  processes.  The  analogical  use  of  the  "sponge" 
concept  implies  that  a complete  analysis  will  require 
techniques  for  representing  and  modifying  models  drawn  from 
other  domains.  The  process-relationship  example  implies 
that  representation  of  general  process  relationships  like 
"inverse  processes",  "feedback  system",  and  "cyclical 
process"  will  have  to  be  included.  There  also  seem  to  be 
multiple  ways  of  describing  what  appears  to  be  essentially 
the  same  information.  These  different  ways  may  be 
generative  in  nature,  but  they  emphasize  different  aspects 
of  the  processes.  For  example,  there  is  the  energy 
viewpoint  from  which  various  processes  appear  to  add  or 
remove  types  of  energy  to  different  actors.  There  is  the 
change-o f-state  viewpoint,  from  which  various  actors  appear 
to  change  form  and  location  as  time  progresses.  These 
multiple  representational  viewpoints  are  different  but  must 
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interact  in  order  to  provide  a complete  representation  of  a 
physical  process.  We  believe  that  defining  them  will 
provide  additional  insights  into  the  nature  of  tutorial 
skill . 
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